Techniques for Command Validation for Access to a Storage Device by a Remote Client

ABSTRACT

Examples are disclosed for access to a storage device maintained at a server. In some examples, a network input/output device coupled to the server may allocate, in a memory of the server, a buffer, a doorbell, and a queue pair accessible to a client remote to the server. For these examples, the network input/output device may assign an Non-Volatile Memory Express (NVMe) namespace context to the client. For these examples, indications of the allocated buffer, doorbell, queue pair, and namespace context may be transmitted to the client. Other examples are described and claimed.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of, claims the benefit of andpriority to previously filed U.S. patent application Ser. No. 13/997,996filed Jun. 25, 2013, entitled “Techniques for Command Validation forAccess to a Storage Device by a Remote Client”, which is a nationalstage application claiming the benefit of and priority to InternationalApplication No. PCT/US2013/021759 entitled “TECHNIQUES FOR COMMANDVALIDATION FOR ACCESS TO A STORAGE DEVICE BY A REMOTE CLIENT” filed Jan.16, 2013, which claims priority to U.S. Provisional Patent ApplicationNo. 61/587,541, filed on Jan. 17, 2012, entitled “TECHNIQUES FOR REMOTESTORAGE USING CIRCUITRY”, which are hereby incorporated by reference intheir entireties.

BACKGROUND

In an example conventional computing arrangement, a client and a serverinclude respective network interface controllers (NICs) or network (NW)input/output (I/O) devices that are capable of communicating with eachother using a Remote Direct Memory Access (RDMA) protocol. The serverincludes a host processor that executes the server's operating systemand associated drivers. The server may also include a storage controllerthat manages access to storage maintained at or by the server. Theclient's NW I/O device issues requests to the server's NW I/O device towrite data to and read data from the storage maintained by the server.The server's operating system, associated drivers, and host processorprocess the requests received by the server's NW I/O device, and issuescorresponding requests to the storage controller. The storage controllerreceives and executes these corresponding requests. After executing thecorresponding requests, the storage controller issues request completioninformation (and associated data if data has been read from the storage)to the server's operating system and associated drivers. From this, theserver's operating system, associated drivers, and host processorgenerate corresponding request completion information and associateddata, and issue the corresponding request completion information andassociated data to the server's NW I/O device. The server's NW I/Odevice then issues the corresponding request completion information andassociated data to the client's NW I/O device.

Thus, in the foregoing conventional arrangement, the server's operatingsystem, associated drivers, and host processor process requests receivedby the server's NW I/O device, and the completion information and datafrom the storage. This may consume substantial amounts of operatingsystem and host processor processing bandwidth. It may also increase theamount of energy consumed and heat dissipated by the host processor.Furthermore, it may increase the latency involved in processing therequests issued by the client's NW I/O device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a first example system.

FIG. 2 illustrates an example completion queue element.

FIG. 3 illustrates a second example system.

FIG. 4 illustrates an example allocation scheme for NVMe Namespaces.

FIG. 5 illustrates a first example bounce buffer allocation.

FIG. 6 illustrates a second example bounce buffer allocation.

FIG. 7 illustrates an example Namespace context index.

FIG. 8 illustrates an example communication flow.

FIG. 9 illustrates an example read command format.

FIG. 10 illustrates an example write and compare command format.

FIG. 11 illustrates an example flush and write uncorrectable commandformat.

FIG. 12 illustrates an example NVMe command completion format.

FIG. 13 illustrates an example NVMe command validation table.

FIG. 14 illustrates an example block diagram for an apparatus.

FIG. 15 illustrates an example of a logic flow.

FIG. 16 illustrates an example of a storage medium.

FIG. 17 illustrates an example network input/output device.

DETAILED DESCRIPTION

As contemplated in the present disclosure, substantial amounts ofoperating system and host processor processing bandwidth may be consumedin a conventional arrangement between a client and a server when theclient attempts to access storage maintained by the server. Recently,servers are including both NW I/O devices and storage controllers havingenhanced capabilities that try to minimize operating system and hostprocessor involvement. For example, hardware elements such as commandsubmission and command completion queues may be utilized by a server'sNW I/O device and storage controllers to enable a remote client toaccess storage via a process known as remote direct memory access(RDMA).

Storage controllers are also being designed to operate in compliancewith relatively new interconnect communication protocols that may workwell with RDMA. Further, these storage controllers may control access tohard disk drives (HDDs) solid state drives (SSDs). The SSDs may include,but are not limited to, various types of non-volatile memory such as3-dimensional cross-point memory, flash memory, ferroelectric memory,silicon-oxide-nitride-oxide-silicon (SONOS) memory, polymer memory,nanowire, ferroelectric transistor random access memory (FeTRAM orFeRAM), nanowire or electrically erasable programmable read-only memory(EEPROM). In some examples, access to HDDs or SSDs may include use ofinterconnect communication protocols described in industry standards orspecifications (including progenies or variants) such as the PeripheralComponent Interconnect (PCI) Express Base Specification, revision 3.0,published in November 2010 (“PCI Express” or “PCIe”) and/or theNon-Volatile Memory Express (NVMe) Specification, revision 1.1,published in October 2012.

Storage controllers that operate in compliance with the NVMeSpecification (“NVMe controllers”) may be capable of minimizingoperating system and host processor involvement when allowing a remoteclient to access storage such as an SSD or an HDD. These types of NVMecontrollers may not have built-in security checks to control access tothe SSD or the HDD by the client. Typically, access checks are handledby the operating system and this may defeat or reduce possibleefficiencies gained through an RDMA process. However, NW I/O devices mayhave built-in security checks and may have a capability to validate aclient to make sure requests for access to the SSD by the client arepermitted or are valid. It is with respect to these and other challengesthat the examples described herein are needed.

In some examples, techniques associated with command validation foraccess to a storage device by a remote client may be implemented. Forthese examples, circuitry for a NW I/O device at a server may be capableof supporting one or more modules associated with receiving, validatingand indicating a status for a received command for the client to accessa storage device controlled by an NVMe controller maintained at theserver. For example, a command for the client to access the storagedevice may be received at the NW I/O device. Features and/or logicexecuted by the circuitry for the NW I/O device may validate the commandbased on whether an operation code (Opcode) indicated in the command issupported by the NW I/O device. The command may also be validated basedon whether an index identifier indicated in the command associates theclient with an NVMe Namspace Identifier allocated to the client. Thefeatures and/or logic executed by the circuitry may then indicate astatus of the received command to the client as one of an invalidcommand (e.g., not authorized) or a completed command (e.g., access wasgranted and command was executed by the NVMe controller).

FIG. 1 illustrates a first example system. As shown in FIG. 1 the firstexample system includes a system 100 having a client 10 that iscommunicatively coupled, via network 50, to server 20. According to someexamples, the terms “host computer,” “host,” “server,” “client,”“network node,” and “node” may be used interchangeably, and may mean,for example, without limitation, one or more end stations, mobileinternet devices, smart phones, media devices, input/output (I/O)devices, tablet computers, appliances, intermediate stations, networkinterfaces, clients, servers, and/or portions thereof. Although client10, server 20, and network 50 will be referred to in the singular, itshould be understood that each such respective component may comprise aplurality of such respective components without departing from theseexamples. According to some examples, a “network” may be or comprise anymechanism, instrumentality, modality, and/or portion thereof thatpermits, facilitates, and/or allows, at least in part, two or moreentities to be communicatively coupled together. Also in some examples,a first entity may be “communicatively coupled” to a second entity ifthe first entity is capable of transmitting to and/or receiving from thesecond entity one or more commands and/or data. Also, data andinformation may be used interchangeably, and may be or comprise one ormore commands (for example one or more program instructions), and/or oneor more such commands may be or comprise data and/or information. Alsofor these examples, an “instruction” may include data and/or one or morecommands.

Client 10 may include remote direct memory access (RDMA)-enabled networkinterface controller (RNIC) herein referred to as network (NW) I/Odevice 106 and/or one or more (and in the example shown in FIG. 1, aplurality of) buffers 13.

As shown in FIG. 1, server 20 may include one or more integrated circuit(IC) chips 180, memory 21, and/or storage 150. One or more chips 180 mayhave circuitry 118 that may include, a NW I/O device such as NW I/Odevice 108 and/or an NVMe controller 112. Although not shown in FIG. 1,in some examples NW I/O device 108 may be a separately attachable devicethat couples to server 20 and includes its own circuitry as describedfurther below.

Also as shown in FIG. 1, the one or more chips 180 that may beincorporated within one or more multi-core host processors (HP) and/orcentral processing units (CPU) 12. Although not shown in the Figures,server 20 also may comprise one or more chipsets or devices to include,but not limited to memory or input/output controller circuitry). NW I/Odevice 108, NVMe controller 112, and/or HP/CPU 12 may be capable ofcommunicating with each other. Additionally, NW I/O device 108, NVMecontroller 112, and/or HP/CPU 12 may be capable of accessing and/orcommunicating with one or more other components of server 20 (such as,memory 21 and/or storage 150), via one or more such chipsets. In someexamples, client 10 and/or NW I/O device 106 may be remote (e.g.,geographically remote), at least in part, from server 20 and/or NW I/Odevice 108.

According to some examples, “circuitry” may comprise, for example,singly or in any combination, analog circuitry, digital circuitry,hardwired circuitry, programmable circuitry, co-processor circuitry,state machine circuitry, and/or memory that may comprise programinstructions that may be executed by programmable circuitry. Also, insome examples, a processor, HP, CPU, processor core (PC), core, andcontroller each may comprise respective circuitry capable of performing,at least in part, one or more arithmetic and/or logical operations,and/or of executing, at least in part, one or more instructions. Anintegrated circuit chip may include one or more microelectronic devices,substrates, and/or dies. Although not shown in the FIG. 1, server 20 mayhave a graphical user interface system that may include, e.g., arespective keyboard, pointing device, and display system that may permita human user to input commands to, and monitor the operation of, server20 and/or system 100. Also, memory may comprise one or more of thefollowing types of memories: semiconductor firmware memory, programmablememory, non-volatile memory, read only memory, electrically programmablememory, random access memory, flash memory, magnetic disk memory,optical disk memory, and/or other or later-developed computer-readableand/or writable memory.

In some examples, storage 150 may include mass storage 156. For theseexamples, storage 150 may include one or more devices into, and/or fromwhich, data may be stored and/or retrieved, respectively. Also, forthese examples, mass storage may include storage capable of non-volatilestorage of data. For example, mass storage 156 may include, withoutlimitation, one or more non-volatile electro-mechanical, magnetic,optical, and/or semiconductor storage devices. These devices may includehard disk drives (HDDs) or solid state drives (SSDs). The SSDs may havenon-volatile types of memory such as 3-dimensional cross-point memory,flash memory, ferroelectric memory, silicon-oxide-nitride-oxide-silicon(SONOS) memory, polymer memory, nanowire, ferroelectric transistorrandom access memory (FeTRAM or FeRAM), nanowire or electricallyerasable programmable read-only memory (EEPROM).

According to some examples, NVMe controller 112, storage 150 or massstorage 156 may be capable of operating in compliance with the PCIeSpecification and/or the NVMe Specification.

One or more machine-readable program instructions may be stored, atleast in part, in memory 21. In operation of server 20, thesemachine-readable instructions may be accessed and executed by one ormore host processors 12, NW I/O device 108, and/or NVMe controller 112.When executed by one or more HP 12, these one or more machine-readableinstructions may result in one or more operating system environments(OSE) 32 being executed at least in part by one or more HP 12, andbecoming resident at least in part in memory 21. Also when thesemachine-readable instructions are executed by NW I/O device 108 and/orNVMe controller 112, these one or more instructions may result in one ormore command interfaces 110 of NVMe controller 112, one or moredoorbells 192, one or more pointers 202, one or more agents 194, one ormore completion queues 124, and/or one or more submission queues 126being established and/or executed by NW I/O device 108 and/or NVMecontroller 112, and/or becoming resident in memory 21.

According to some examples, one or more OSE 32 may include one or moreoperating systems (OS) 31 and/or one or more NW I/O device and/or NVMecontroller drivers 33. These one or more drivers 33 may be mutuallydistinct from one or more OS 31, at least in part. Alternatively oradditionally, without departing from these examples, one or morerespective portions of one or more OS 32 and/or drivers 33 may not bemutually distinct, at least in part, from each other and/or may beincluded, at least in part, in each other. Likewise, without departingfrom these examples, circuitry 118, NW I/O device 108, and/or NVMecontroller 112 may be distinct from, or alternatively, may be includedin the one or more not shown chipsets and/or HP 12. Also withoutdeparting from these examples, one or more portions of memory 21 may beincluded in or maintained at NW I/O device 108, NVMe controller 112,circuitry 118, HP 12, and/or IC 180.

In some examples, a portion or subset of an entity may include all orless than all of the entity. Also, for these examples, a process,thread, daemon, program, driver, operating system, application, kernel,and/or virtual machine monitor each may (1) include, at least in part,and/or (2) result, at least in part, in and/or from, execution of one ormore operations and/or program instructions.

According to some examples, a command interface may facilitate, permit,and/or implement, at least in part, exchange, transmission, and/orreceipt of data and/or one or more commands. For these examples, aqueue, buffer, and/or doorbell may be one or more locations (e.g.,specified and/or indicated, at least in part, by one or more addresses)in memory in which data and/or one or more commands may be stored, atleast temporarily. Also, a queue element may include data and/or one ormore commands to be stored and/or stored in one or more queues, such as,for example, one or more descriptors and/or one or more commands.Additionally, for these examples, a pointer may indicate, address,and/or specify, at least in part, one or more locations and/or one ormore items in memory.

In some examples, NW I/O device 106 and NW I/O device 108 may exchangedata and/or commands via network 50 in accordance with one or moreprotocols that may comply and/or be compatible with an RDMA protocolsuch as Internet Wide Area RDMA protocol (iWARP), Infiniband (IB)protocol, Ethernet protocol, Transmission Control Protocol/InternetProtocol (TCP/IP) protocol and/or RDMA over Converged Ethernet (RoCE)protocol. For example, the iWARP protocol may comply and/or becompatible with Recio et al., “An RDMA Protocol Specification,” InternetDraft Specification, Internet Engineering Task Force (IETF), 21 Oct.2002. Also for example, the Ethernet protocol may comply and/or becompatible with Institute of Electrical and Electronics Engineers, Inc.(IEEE) Std. 802.3-2008, Dec. 26, 2008. Additionally, for example, theTCP/IP protocol may comply and/or be compatible with the protocolsdescribed in Internet Engineering Task Force (IETF) Request For Comments(RFC) 791 and 793, published September 1981. Also, the IB protocol maycomply and/or be compatible with Infiniband Architecture Specification,Vol. 2, Rel. 1.3, published November 2012. Additionally, for example,the RoCE protocol may comply and/or be compatible with Supplement toInfiniband Architecture Specification, Vol. 1, Rel. 1.2.1, Annex A16:“RDMA over Converged Ethernet (RoCE)”, published April 2010. Manydifferent, additional, and/or other protocols may be used for such dataand/or command exchange without departing from these examples (e.g.,earlier and/or later-developed versions of the aforesaid, related,and/or other protocols).

According to some examples, circuitry 118 may permit and/or facilitate,at least in part, NW I/O device 106's access, via NW I/O device 108, ofone or more command interfaces 110. For example, circuitry 118 maypermit and/or facilitate, at least in part, NW I/O device 106 being ableto so access one or more command interfaces 110 in a manner that isindependent of OSE 32 in server 20. This accessing may include, forexample, the writing of at least one queue element (e.g., one or morequeue elements (QE) 116) to one or more submission queues 114 in one ormore command interfaces 110. This may command NVMe controller 112 toperform, at least in part, one or more operations involving storage 150and/or mass storage 156 associated with NVMe controller 112. NVMecontroller 112 may perform these one or more operations in response, atleast in part, to the one or more queue elements 116 (e.g., after and inresponse, at least in part, to the one or more queue elements 116 beingwritten into one or more submission queues 114). These one or moreoperations involving storage 150 and/or mass storage 156 may compriseone or more write operations and/or one or more read operationsinvolving, at least in part, storage 150 and/or mass storage 156. Forthese examples, client 10 thus may be able to access storage 150 and/ormass storage 156 via the one or more read operations and/or one or morewrite operations executed by NVMe controller 112.

By way of example, in operation of system 100, client 10 and/or NW I/Odevice 106 may authenticate client 10 and/or NW I/O device 106 to server20 and/or logic and/or features at NW I/O device 108. This may result inclient 10 and/or NW I/O device 106 being granted permission to access,at least in part, devices maintained at or controlled by elements ofserver 20 (e.g., via NW I/O device 108). Contemporaneously, after, orprior to this, at least in part, NW I/O device 108, NVMe controller 112,one or more agents 194, and/or OSE 32 may generate, establish, and/ormaintain, at least in part, in memory 21, one or more interfaces 110and/or one or more indicators 181 that may indicate, at least in part,where (e.g., one or more locations) in memory 21 one or more interfaces110 and/or the components thereof may be located. For example, one ormore indicators 181 may indicate, at least in part, one or morelocations in memory 21 where one or more submission queues 114, one ormore completion queues 120, one or more doorbells 170, and/or one ormore buffers 130A . . . 130N may be located. NW I/O device 108 mayprovide, via network 50, one or more indicators 181 to NW I/O device106. Thereafter, NW I/O device 106 may use one or more of the one ormore indicators 181 to access one or more command interfaces 110 and/orone or more components of the one or more command interfaces 110. One ormore indicators 181 may be or comprise, at least in part, one or morehandles (e.g., assigned to transaction contexts) for one or more regionsin memory 21, such as, in this embodiment, one or more service tags(STags) or transaction tags (TTags) that may comply and/or may becompatible with an RDMA (e.g., iWARP, IB, RoCE) protocol. In someexamples, the one or more regions in memory 21 may be included in one ormore bounce buffers maintained to facilitate remote access of storage150 or mass storage 156 by client 10.

After receiving one or more indicators 181, client 10 and/or NW I/Odevice 106 may issue one or more commands 105 to server 20, via network50 and NW I/O device 108, to NVMe controller 112 in a manner thatby-passes and/or is independent of the involvement of OSE 32. The one ormore commands 105 may command NVMe controller 112 to perform one or moreoperations involving storage 150 and/or mass storage 156.

According to some examples, one or more commands 105 may comply and/orbe compatible with an RDMA (e.g., iWARP, IB, RoCE) protocol. One or morecommands 105 may include and/or specify, at least in part, one or morequeue elements 116 that may embody and/or indicate, at least in part,the one or more operations involving storage 150 and/or mass storage 156that are being commanded. Although not shown in FIG. 1, one or morecommands 105 may comprise, specify, and/or indicate, at least in part,one or more of the indictors 181 that may indicate one or more locationsin one or more submission queues 114 as one or more intendeddestinations of one or more queue elements 116.

In some examples, one or more queue elements 116 may command that NVMecontroller 112 perform one or more write operations involving storage150 and/or mass storage 156. Therefore, one or more commands 105 alsomay include and/or specify, at least in part, data 199 to be written, asa result of executing one or more queue elements 116, by NVMe controller112 to storage 150 and/or mass storage 156. One or more commands 105 mayinclude, specify, and/or indicate, at least in part, one or more of theindicators 181 that may indicate one or more locations of one or morebuffers (e.g., buffer(s) 13) to which data 199 is to be written (atleast temporarily) to a client 10.

In response, at least in part, to receipt of one or more commands 105,NW I/O device 108 may directly write (e.g., in accordance with RDMA(e.g., iWARP, IB, RoCE) protocol and/or in a manner that by-passesand/or is independent of OSE 32), in the manner commanded by one or morecommands 105, one or more queue elements 116 and data 199 to one or moresubmission queues 114 and one or more buffers 130A, respectively. Thus,in effect, by issuing one or more commands 105 to NW I/O device 108, NWI/O device 106 may write one or more queue elements 116 and data 199 toone or more submission queues 114 and one or more buffers 130A,respectively.

One or more commands 105 also may comprise and/or specify one or morevalues 201 and one or more of the indicators 181 that may indicate oneor more locations of one or more doorbells 170 to which one or morevalues 201 may be written. In response, at least in part, to these oneor more values 201 and these one or more of the indicators 181 in one ormore commands 105, NW I/O device 108 may directly write (e.g., inaccordance with RDMA (e.g., iWARP, 1B, RoCE) protocol and/or in a mannerthat by-passes and/or is independent of OSE 32), in the manner commandedby one or more commands 105, one or more values 201 in doorbell 170. Thewriting of one or more values 201 in doorbell 170 may ring doorbell 170.Thus, in effect, by issuing one or more commands 105 to NW I/O device108, NW I/O device 106 may ring doorbell 170.

According to some examples, the ringing of a doorbell that is associatedwith an entity may comprise and/or involve, at least in part, thewriting one or more values to one or more memory locations (e.g.,associated with, comprising, and/or embodying the doorbell) that mayresult in and/or trigger, at least in part, the entity performing, atleast in part, one or more operations and/or actions. In some examples,the doorbells 170 and/or 192 may appear to CPU 12 and/or server 20 asone or more respective memory locations (not shown) in respective memory(not shown) in NVMe controller 112 and/or NW I/O device 108,respectively.

In response, at least in part, to the ringing of doorbell 170, NVMecontroller 112 may return to a fully operational state (e.g., if NVMecontroller 112 had previously entered a reduced power state relative tothis fully operational state), and may read one or more queue elements116 that were written into one or more submission queues 114. NVMecontroller 112 may then execute, at least in part, the one or morecommands that are specified and/or embodied by one or more queueelements 116. This may result in NVMe controller 112 performing, atleast in part, the one or more operations (e.g., one or more writes tostorage 150 and/or mass storage 156 of data 199 stored in one or morebuffers 130A) involving storage 150 and/or mass storage 156.

After completion, at least in part, of these one or more operationsinvolving storage 150 and/or mass storage 156, NVMe controller 112 maygenerate and write, at least in part, one or more completion queueelements (CQE) 129 to one or more completion queues 124. Also aftercompletion, at least in part, of these one or more operations involvingstorage 150 and/or mass storage 156, NVMe controller 112 also may write,at least in part, one or more values to one or more doorbells 192associated with NW I/O device 108. This may ring one or more doorbells192. In response, at least in part, to the ringing of one or moredoorbells 192, NW I/O device 108 may write (e.g., via one or more RDMAwrite operations) one or more completion queue elements 190 to one ormore completion queues 120 and then forward the one or more completionqueue elements 190 to one or more buffers 13 in client 10 (e.g., via oneor more responses 197).

After one or more (e.g., several) such write and/or read operationsinvolving storage 150 and/or mass storage 156 have been performed, atleast in part, one or more agents 194 may carry out certain managementfunctions. For example, one or more agents 194 may establish, at leastin part, one or more submission queue entries/elements (E) 196A . . .196N in one or more submission queues 126 associated with NW I/O device108 and/or one or more submission queue entries/elements QE A . . . QE Nin table 250 (see FIG. 2). As is discussed below, these elements 196A .. . 196N and/or QE A . . . QE N, when executed, at least in part, by NWI/O device 108, may permit and/or facilitate copying or forwarding, atleast in part, of one or more other queue entries (e.g., one or moreNVMe controller 112 completion entries 190) to client 10 and/or NW I/Odevice 106 and/or data read by NVMe controller 112.

These management functions also may include the updating (e.g.,appropriately advancing), at least in part, by one or more agents 194 ofone or more pointers (e.g., ring pointers PNTR 202) associated with oneor more queue pairs (e.g., submission/completion queue pair 114, 120and/or submission/completion queue pair 126, 124) associated with the NWI/O controller 108 and the NVMe controller 112. This may permit newentries to the queue pairs to be stored at locations that will notresult in erroneous overwriting of other entries in the queue pairs.Additionally, as part of these management functions, the one or moreagents 194 may indicate one or more of the buffers 130A . . . 130N thatmay be available to be reused.

As another example, one or more queue elements 116 may command that NVMecontroller 112 perform one or more read operations involving storage 150and/or mass storage 156. Therefore, one or more commands 105 also mayinclude and/or specify, at least in part, one or more locations (e.g.,Namespaces) in storage 150 and/or mass storage 156 from which NVMecontroller 112 is to read data 199, as a result of executing one or morequeue elements 116.

In response, at least in part, to receipt of one or more commands 105,NW I/O device 108 may directly write (e.g., in accordance with an RDMA(e.g., iWARP, IB, RoCE) protocol and/or in a manner that by-passesand/or is independent of OSE 32), in the manner commanded by one or morecommands 105, one or more queue elements 116 to one or more submissionqueues 114. Thus, in effect, by issuing one or more commands 105 to NWI/O device 108, NW I/O device 106 may write one or more queue elements116 to one or more submission queues 114 and one or more buffers 130A,respectively.

In this example, one or more commands 105 also may comprise and/orspecify one or more values 201 and one or more of the indicators 181that may indicate one or more locations of one or more doorbells 170 towhich one or more values 201 are to be written. In response, at least inpart, to these one or more values 201 and these one or more of theindicators 181 in one or more commands 105, NW I/O device 108 maydirectly write (e.g., in accordance with an RDMA (e.g., iWARP, IB, RoCE)protocol and/or in a manner that by-passes and/or is independent of OSE32), in the manner commanded by one or more commands 105, one or morevalues 201 in doorbell 170. The writing of one or more values 201 indoorbell 170 may ring doorbell 170. Thus, in effect, by issuing one ormore commands 105 to NW I/O device 108, NW I/O device 106 may ringdoorbell 170.

In response, at least in part, to the ringing of doorbell 170, NVMecontroller 112 may return to a fully operational state (e.g., if NVMecontroller 112 had previously entered a reduced power state relative tothis fully operational state), and may read one or more queue elements116 that were written into one or more submission queues 114. NVMecontroller 112 then may execute, at least in part, the one or morecommands that are specified and/or embodied by one or more queueelements 116. This may result in NVMe controller 112 performing, atleast in part, the one or more operations (e.g., one or more reads ofstorage 150 and/or mass storage 156 to obtain data 199) involvingstorage 150 and/or mass storage 156 and storing data 199 in one or morebuffers (e.g., one or more buffers 130A).

After completion, at least in part, of these one or more operationsinvolving storage 150 and/or mass storage 156, NVMe controller 112 maygenerate and write, at least in part, one or more completion queueelements 129 to one or more completion queues 124. Also aftercompletion, at least in part, of these one or more operations involvingstorage 150 and/or mass storage 156, NVMe controller 112 also may write,at least in part, one or more values to one or more doorbells 192associated with NW I/O device 108. This may ring one or more doorbells192. In response, at least in part, to the ringing of one or moredoorbells 192, NW I/O device 108 may obtain queue elements 129 from theone or more completion queues 124 and forward or write one or morecompletion queue elements 190 to one or more completion queues 120 tofacilitate the transfer of data 199 (e.g., via one or more RDMA writeoperations with NW I/O device 106) to one or more buffers 13 in client10 (e.g., via one or more responses 197).

According to some examples, command interface 110 may be asynchronous inthat, for example, completion queue elements may not be stored in anorder in one or more completion queues 120 that corresponds to (1) theorder in which command queue elements are stored in the one or moresubmission queues 114, (2) the order in which such command queueelements are executed and/or completed by the NVMe controller 112,and/or (3) the order in which completion queue elements 190 are storedin one or more completion queues 120 and/or provided to NW I/O device106 and/or client 10. In operation, NW I/O device 106 and/or client 10may appropriately reorder, in the case of write commands issued from theclient 10 and/or NW I/O device 106, corresponding completion queueelements 190 received from NW I/O device 108. However, in the case ofread commands, in this embodiment, in order to permit respective dataread from storage 150 and/or mass storage 156 to be appropriatelyassociated with corresponding completion queue elements for transmissionto client 10 and/or NW I/O device 106, each completion queue element(e.g., completion queue element 190) resulting from completionindications placed in completion queues 120 by NW I/O device 108 mayinclude the elements illustrated in FIG. 2.

As shown in FIG. 2, completion queue element (e.g., completion queueelement 190) may include one or more command parameters 304, one or morecommand queue identifiers 306, one or more command queue head positionindicators 308, status information 310, one or more queue phase bit (P)312, and/or one or more command identifiers 302. One or more commandparameters 304 may be and/or indicate one or more command specificparameters of the one or more queue elements 116 and/or commands 105that may correspond to and/or be associated with the one or morecompletion queue elements 190. One or more command queue identifiers 306may indicate and/or specify the one or more submission queues 114 towhich the one or more queue elements 116 were written. One or morecommand queue head position indicators 308 may indicate the currentposition (e.g., in the one or more submission queues 114 identified byone or more command queue identifiers 306) at which the one or morequeue elements 116 may be located. Status information 310 may indicatewhether the one or more commands 105 and/or one or more queue elements116 were successfully performed by the NVMe controller 112. One or morephase bits 312 may indicate whether the one or more completion queueelements 190 constitute the most recently added valid entry (e.g., toservice) in one or more completion queues 120. One or more commandidentifiers 302 may indicate, at least in part, and/or be identical toone or more corresponding command identifiers in the corresponding oneor more queue elements 116. Command identifiers 302 may permit one ormore completion queue elements 190 to be correctly associated with oneor more corresponding queue elements 116 and/or with the respective data199 read from the storage 150 and/or mass storage 156 as a result of theexecution of these one or more corresponding queue elements 116.

In some examples, one or more command identifiers 302 may be selected soas not to collide with and/or be identical to any other commandidentifiers that may be currently used by any completion queue elementsthat have not yet been provided to client 10 and/or NW I/O device 106 byNW I/O device 108. The command identifiers that may be used in system100 may be pre-calculated and/or pre-generated, and may be used asrespective indices INDEX A . . . INDEX N for respective entries ENTRY A. . . ENTRY N in a table 250 that may be stored, at least in part, inmemory 21. Each of the entries ENTRY A . . . ENTRY N in the table 250may store one or more respective pre-calculated and/or pre-generatedcommand queue elements QE A . . . QE N that may be associated with NWI/O device 108. Each respective element QE A . . . QE N may beassociated with one or more respective buffers in one or more buffers130A . . . 130N. Each of the buffers in one or more buffers 130A . . .130N into which NVMe controller 112 may store data read from storage 150and/or mass storage 156 also may be associated with one or morerespective submission identifiers used in system 100 and/or respectiveentries ENTRY A . . . ENTRY N.

The command queue elements QE A . . . QE N may be stored and/ormaintained in table 250 by client 10 and/or one or more agents 194. Ifone or more buffers 130A . . . 130N are statically allocated, table 250may be static, and may correspond in terms of, for example, allocationcharacteristics to one or more buffers 13 that may be allocated in theclient 10.

By way of example, after NVMe controller 112 reads data 199 from storage150 and/or mass storage 156, NVMe controller 112 may store the data 199in one or more buffers (e.g., one or more buffers 130A) that may beassociated with one or more command identifiers 302, and may store oneor more queue elements 129 in one or more completion queues 124. Inresponse, at least in part, to NVMe controller 112 ringing one or moredoorbells 192, NW I/O device 108 may determine, based at least in partupon one or more queue phase bits 312, the one or more most recentlyadded valid completion queue in one or more completion queues 120 or124. NW I/O device 108 may use the one or more command identifiers 302in one or more completion queue elements 190 to index into table 250 tolocate the one or more entries (e.g., one or more entries ENTRY A) andone or more command queue elements (e.g., one or more queue elements QEA) in table 250 that may be associated with and/or identified, at leastin part, by one or more command identifiers 302. NW I/O device 108 mayexecute, at least in part, one or more commands that may be associatedwith and/or embodied by these one or more command queue elements QE A.This may result, at least in part, in NW I/O 108 reading one or morebuffers 130A to obtain data 199, and transmitting data 199 and one ormore completion queue elements 190 to NW I/O device 106 and/or client 10(e.g., via one or more responses 197). As a result, data 199 and/or oneor more completion queue elements 190 may be copied into one or moreclient buffers 13.

Alternatively, in some examples, NW I/O device 108 may comprise, atleast in part, a state machine (not shown). This state machine may beindependent and/or separate from, at least in part, of one or moresubmission queues 126 that may be associated with and/or utilized by NWI/O device 108. This state machine may locate one or more command queueelements QE A in table 250 based at least in part upon one or morecommand identifiers 302, and may copy the one or more queue elements QEA into one or more corresponding submission queue elements 196A in oneor more submission queues 126. The state machine then may signal NW I/Odevice 108 to access and execute, at least in part, one or moresubmission queue elements 196A in one or more submission queues 126.

Further alternatively, without departing from these examples, prior tocompleting one or more read operations involving storage 150 and/or massstorage 156, NVMe controller 112 may locate and/or select, at least inpart, one or more queue elements QE A in and/or from table 250, based atleast in part upon one or more command identifiers 302. NVME controller112 then may write into one or more completion queue elements 190 intoone or more completion queues 120, and may write one or more queueelements QE A into one or more corresponding submission queue elements196A in one or more submission queues 126. NVME controller 112 then mayring one or more doorbells 192. This may result in NW I/O device 108accessing and executing, at least in part, one or more submission queueelements 196A in one or more submission queues 126. This may result, atleast in part, in NW I/O device 108 reading one or more buffers 130A toobtain data 199, and transmitting data 199 and one or more completionqueue elements 190 to NW I/O device 106 and/or client 10 (e.g., via oneor more responses 197). As a result, data 199 and/or one or morecompletion queue elements 190 may be copied into one or more clientbuffers 13.

In this alternative, firmware and/or one or more agents 194 executed, atleast in part, by NVMe controller 112 may maintain per-queue-paircontext information to indicate one or more queue pairs used for RDMAtransactions. This context information also may include various pointers(e.g., to one or more arrays of submission queue elements 196A . . .196N to move data from one or more buffers 130A . . . 130N to one ormore buffers 13, and/or the head of one or more submission queues 126),one or more locations of one or more doorbells 192 and one or morevalues to ring the one or more doorbells 192, and/or local copies ofhead and/or pointers to the one or more submission queues 126. Variousof these pointers (e.g., the head and tail pointers) may be dynamicallyupdated by firmware executed by NVMe controller 112.

Additionally or alternatively, without departing from these examples, NWI/O device 108 and/or NVMe controller 112 may be comprised, at least inpart, in the not shown chipset, or in a not shown circuit board ordevice. Also additionally or alternatively, without departing from thisembodiment, storage 150 and/or mass storage 156 may be comprised, atleast in part, internally in server 20 or be external to server 20.

Further although the foregoing description has been made with referenceto NW I/O device 108 being an RNIC, and NVMe controller 112 being anNVMe compliant storage controller, the principles of this embodiment maybe applied to circumstances in which protocols other than and/or inaddition to RDMA or NVMe may be employed, and/or in which NVMecontroller 112 may be involved in executing and/or facilitatingoperations that do not involve storage 150 (e.g., other and/oradditional input/output and/or communication-related operations).Accordingly, without departing from the above mentioned examples, NW I/Odevice 108 may utilize, and/or communications between client 10 andserver 20 may employ, protocols other than and/or in addition to RDMA.Also, without departing from this embodiment, NW I/O device 108 or NVMecontroller 112 may be involved in executing and/or may facilitateexecution of such other and/or additional operations that may employprotocols other than PCIe or NVMe protocols. In these additional and/oralternative arrangements, hardware and/or firmware circuitry (not shown)may be comprised in circuitry 118 that may permit, at least in part,writing to doorbells 170 and/or 192 via, e.g., one or more interruptmechanisms (e.g., one or more message signaled interrupts (MSI/MSI-X)and/or other mechanisms). This embodiment should be viewed broadly ascovering all such modifications, variations, and alternatives.

Thus, in some examples, circuitry may be arranged, at least in part, toenable a first NW I/O device in a client to access, via a second NW I/Odevice in a server that is remote from the client and in a manner thatis independent of an operating system environment in the server, atleast one command interface of another (e.g., storage, and/oranother/additional type of) controller of the server. The NW I/O devicein the client and the NW I/O device in the server may be or compriserespective remote direct memory access-enabled network interfacecontrollers (e.g., controllers capable, at least in part, of utilizingand/or communicating via RDMA). The command interface may include atleast one (e.g., storage, and/or other/additional type of) controllercommand queue. Such accessing may include writing at least one queueelement to the at least one submission queue to command the anothercontroller to perform at least one operation (e.g., involving storage,and/or involving one or more other and/or additional types ofoperations, such as, other and/or additional input/output operations)associated with the another controller (e.g., an NVMe controller). Theanother controller may perform the at least one operation in response,at least in part, to the at least one queue element. Many alternatives,variations, and modifications are possible.

Thus, in some examples, the one or more command interfaces 110 of NVMecontroller 112 in server 20 may be directly accessed by the client's NWI/O device 106 via one or more RDMA transactions, in a manner thatby-passes, is independent of, and/or does not involve the server's OSE32 and/or CPU 12. Advantageously, this may permit storage commands,data, and completion messages to be communicated between the client andserver much more quickly and efficiently, and with reduced latency.Furthermore, in this embodiment, interactions between NW I/O device 108and NVMe controller 112 may be carried out entirely or almost entirelyby hardware (e.g., utilizing peer-to-peer memory and doorbell writes),and also in a manner that by-passes, is independent of, and/or does notinvolve the server's OSE 32 and/or CPU 12. Advantageously, this maypermit such interactions to be carried out much more quickly andefficiently, and with reduce latency. Additionally, the above featuresof this embodiment may reduce the server's power consumption, heatdissipation, and the amount of bandwidth consumed by the OSE 32 and CPU12.

Many other modifications are possible. For example, as statedpreviously, in this embodiment, client 10 may comprise a plurality ofclients. If RDMA is employed for communications between server 20 andthe clients 10, in this embodiment, advantageously, the clients 10 maydynamically share buffers 130A . . . 130N, as a common pool of buffers,between or among the client 10 in carrying out their communications withserver 20, NW I/O device 108, and/or NVMe controller 112. In order topermit such buffer sharing, NW I/O device 108 may be capable ofmanipulating, adjusting, and/or modifying, at least in part,buffer-specifying information that may be indicated, at least in part,in commands 105 provided to the server 20 by the clients 10 in order toallow the buffers 130A . . . 130N and/or other server resources to beshared among the clients 10 without resulting in, for example,contention-related issues.

For example, the one or more indicators 181 and/or STags/TTags indicatedby the one or more indicators 181 may include respective informationthat NW I/O device 108 may associate with one or more buffers and/orbuffer pools in the buffers 130A . . . 130N, instead of and/or inaddition to one or more memory region handles. In this arrangement, theclients 10 may perform RDMA read operations utilizing such indicators181 and NW I/O device 108 may perform write operations to the one ormore buffers and/or buffer pools indicated by the respective informationand/or indicators 181. In carrying out its operations, NW I/O device 108may appropriately adjust the actual commands and/or command queueelements provided to NVMe controller 112 in order to result in thecorrect buffers, etc. being written to by NVMe controller 112 when NVMecontroller 112 carries out such commands and/or command queue elements.

Alternatively or additionally, without departing from the aboveexamples, NW I/O device 108 may include and/or be associated with ashared receive queue (not shown) to receive, for example, commands 105from multiple clients 10. NW I/O device 108 may be capable ofsubstituting, at least in part, one or more appropriate server bufferaddresses, values, and/or other information into one or more portions(e.g., queue elements 116, values 201, indicators 181, and/or otherinformation) of the received commands 105 to permit sharing of thestructures in the one or more command interfaces 110 between or amongmultiple clients 10, without resulting in contention or otherdegradation in performance. In this arrangement, the clients may not beprovided and/or utilize one or more STags to the storage controller'scommand queue and/or doorbell, and writing to these structures may beperformed by the server's NW I/O device 108. Advantageously, this maypermit multiple clients 10 that may be associated with and/or utilizethe shared receive queue to utilize and/or share, at least in part, thesame storage controller command queue, doorbell, and/or otherstructures.

For example, in the case of a write operation, one or more indicators181, one or more values 201, and/or other information in one or morecommands 105 may indicate, at least in part, one or more storagecontroller STags or TTags for the write operation (and relatedinformation), and/or one or more RDMA STags or TTags to one or morebuffers to which one or more completion queue elements may be written.Based at least in part upon the one or more received commands 105 and/orother information stored in NW I/O device 108, NW I/O device 108 mayselect one or more buffers in buffers 130A . . . 130N and one or morelocations in the submission queue 114 to which to post the data 199 tobe written and one or more corresponding command queue elements to NVMecontroller 112, respectively. NW I/O device 108 may post the data 199and the one or more corresponding command queue elements in accordancewith such selections, and thereafter, may ring doorbell 170. As postedby NW I/O device 108, the one or more command queue elements mayindicate the one or more storage controller STags or TTags supplied inthe one or more commands 105, command identifier 302, security contextinformation (e.g., to permit validation of the one or more storagecontroller STags or TTags), and/or one or more STags/TTags to the one ormore buffers to which data 199 has been posted. After NVMe controller112 has completed, at least in part, the requested one or more writeoperations and posted one or more completion queue elements, NVMecontroller 112 may ring doorbell 192. Based at least in part uponinformation in table 250, NW I/O device 108 may generate and forward tothe one or more clients that provided the received command 105 one ormore appropriate responses 197.

In the case of a read operation, generally analogous information may beprovided in command 105 and generally analogous operations may beperformed by NW I/O device 108 and/or NVMe controller 112. However, inthe case of a read operation, the data 199 read by NVMe controller 112may be stored by NVMe controller 112 to one or more of the buffers 130A. . . 130N specified by the NW I/O device 108, and may be read by the NWI/O device 108, instead of vice versa (e.g., as may be the case in awrite operation). NW I/O device 108 may transmit the read data 199 tothe one or more clients that provided the received command 105 in one ormore responses 197. In the foregoing arrangement, command 105 may besimilar or identical to a command that may be utilized by a client toaccess storage local to the client, at least from the vantage point ofone or more client-executed applications initiating such access.Advantageously, this may permit remote operations and/or RDMAtransactions of the types previously described to be substantiallytransparent to these one or more client-executed applications.

Thus, in some examples, advantageously, it may be possible for multipleclients to share the storage controller's command queue, doorbells,and/or the server's buffers, and/or to write to these structures (viathe server's NW I/O device) using an RDMA protocol, without sufferingfrom resource contention issues (and/or other disadvantages) that mightotherwise occur. The server's NW I/O device may be capable of modifying,at least in part, information associated with and/or comprised in theclients' commands 105 to facilitate such sharing and/or sharing of RDMASTag/TTag information between or among the clients. Advantageously, thismay permit RDMA protocol to be employed for command communication and/orcompletion information between the server and multiple clients, withimproved scalability, while reducing the memory consumption to implementsuch features, and without degradation in communication line rate.

FIG. 3 illustrates a second example system. As shown in FIG. 3, thesecond example system includes a system 300. According to some examples,system 300 may include multiple client nodes 310-1 to 310-n (where “n”represents any positive integer greater than 3) and a server 305. Forthese examples, a NW I/O device 330, an NVMe controller 350 and bouncebuffer(s) 360 may be located with and/or maintained at server 305.

In some examples, logic and/or features executed by circuitry for eithernetwork I/O device 330 and/or server 305 may allocate resources toclients 310-1 to 310-n to facilitate remote access to a storage device(not shown) controlled by NVMe controller 350. For these examples,separate I/O queue pairs (QPs) 320-1 to 320-n and separate NVMe QPs340-1 to 340-n may be allocated or assigned to clients 310-1 to 310-n,respectively. Also, at least portions of bounce buffer(s) 360 may beallocated or assigned to clients 310-1 to 310-n. I/O QPs 320-1 to 320-n,NVME QPs 340-1 to 340-n or bounce buffer(s) 360 may be part of systemmemory resident at server 305. Alternatively, I/O QPs 320-1 to 320-n maybe maintained at or with NW I/O device 330 and NVMe QP 340-1 to 340-nmay be maintained at or with NVMe controller 350.

According to some examples, I/O QPs 320-1 to 320-n may separatelyinclude both command submission queues and command completion queuesutilized by logic and/or features at NW I/O device 330 to exchangeinformation with clients 310-1 to 310-n regarding commands to access thestorage controlled by NVMe controller 350. Also, NVMe QPs 340-1 to 340-nmay separately include command submission queues and command completionqueues utilized by logic and/or features at NW I/O device 330 and atNVMe controller 350 to facilitate the relay of commands from clients310-1 to 310-n to NVMe controller 350. For these examples, NVMe QPs340-1 to 340-n are not directly accessible by clients 310-1 to 310-n.Since NVMe QPs 340-1 to 340-n are not directly accessible to clients310-1 to 310-n, logic and/or features at NW I/O device 330 may becapable of validating commands received from these clients before theyare forwarded or relayed to NVMe QPs 340-1 to 340-1 n.

In some examples, as described more below, logic and/or featuresexecuted by circuitry at NW I/O device 330 may first validate commandsreceived from one or more clients 310-1 to 310-n to access storagecontrolled by NVMe controller 350. The logic and/or features may then becapable of indicating a status of the received command as either invalidor as completed. As part of this example process, the logic and/orfeatures may utilize I/O QPs 320-1 to 320-n, NVMe QP 340-1 to 340-n orbounce buffer(s) 360 to facilitate the access in a similar manner asdescribed above for FIGS. 1 and 2.

FIG. 4 illustrates an example allocation scheme 400 for NVMe Namespaces.According to some examples, logic and/or features maintained at server305 (e.g., server software) may be capable of allocating given NVMeNamespace Identifiers. These given NVMe Namespace Identifiers may thenbe used by logic and/or features at NW I/O device 330 as part of avalidation process to allow RDMA access to at least portions of storagecontrolled by NVMe controller 350. For these examples, the NVMeNamespace Identifiers may be associated with a quantity of non-volatilememory included in the storage that may be formatted into logical blockshaving logical block addresses. As shown in FIG. 4, these NVMe NamespaceIdentifiers are shown as NVME Namespace 401, 402 and 403. Also, NVMeNamespace 401 is shown as being further sub-divided to include 401-1,401-2 and 401-3 while NVMe Namespace 403 is shown including 403-1, 403-2and 403-3.

According to some examples, the logic and/or features maintained atserver 305 may assign one or more Namespace contexts (NSCs) to the NVMeNamespace identifiers. For example, as shown in FIG. 4, NSCs 11, 12 and13 are separately assigned to NVMe Namespace identifiers 401-1, 401-2and 401-3, respectively. Meanwhile, NSC 10 may be assigned to NVMeNamespace identifier 401. Also, as shown in FIG. 4, NSCs 20, 21 and 22are each assigned to the same Namespace Identifier 402. As mentionedmore below, assigned or allocated NSCs may be communicated to clients310-1 to 310-n and may then be used as an index identifier by logicand/or features at NW I/O device 330 to determine whether a given NSCincluded in a command received from a given client associates the givenclient with an NVMe Namespace that has been allocated to that givenclient.

In some examples, the logic and/or features at NW I/O device 330 mayassociate protection domains (PDs) with given I/O QPs. These PDs mayprovide a client with a protected access to one or more Namespaceshaving given Namespace Identifiers. For example, PD=11 includes NSC 11,NSC 21 and NSC 32 that may provide protected access to NamespaceIdentifiers 401-1, 402 and 403-2, respectively. According to someexamples, as shown in FIG. 4, PDs 11, 22, 33 and 44 may be associatedwith I/O QPs 320-1, 320-2, 320-3 and 320-n, respectively. Further, asmentioned above for FIG. 3, I/O QP 320-1 was allocated for use by client310-1. So by associating PD 11 with I/O QP 320-1, client 310-1 may haveprotected access to Namespace Identifiers 401-1, 402 and 403-2. Theother clients may then have protected access according to the PDsassociated with their respectively assigned or allocated I/O QPs.

FIG. 5 illustrates a first example bounce buffer allocation. In someexamples, as shown in FIG. 5, logic block addresses (LBAs) of a bouncebuffer 360-1 may be assigned or allocated to transaction contexts havingtransaction tags (TTags) 510-1 to 510-3. For these examples, the LBAsmay be for system memory maintained at server 305 and may be assigned bylogic and/or features maintained at server 305 (e.g., server software).LBAs assigned to a given transaction context may be used for temporarystorage to facilitate data transfers between clients and storagecontrolled by NVMe controller 350. Also as shown in FIG. 5, metadataareas may be reserved within each of the LBAs allocated to a giventransaction context for metadata. The metadata, for example, may beassociated with commands for access to the storage controlled by NVMecontroller 350 and may include information (e.g., management or controlinformation) associated with data transfers between clients and thestorage.

According to some examples, one or more transaction contexts may beallocated to clients 310-1 to 310-n and those allocations communicatedby the logic and/or features at server 305 as having TTags 510-1, 510-2and 510-3 to both the clients and to the logic and/or features at NW I/Odevice 330. For these examples and as mentioned more below, assigned orallocated TTags may be used by logic and/or features at NW I/O device330 to possibly validate a command received from a client that includesor references an allocated TTag. Also, the amount of memory capacityincluded in the LBAs assigned to a given transaction context may limitthe amount of data that may be transferred for each command receivedfrom the client. If a command includes a request or need for more bouncebuffer memory capacity than assigned to a TTag, that command may bedeemed as invalid and possibly rejected by logic and/or features at NWI/O device 330.

FIG. 6 illustrates a second example bounce buffer allocation. In someexamples, as shown in FIG. 6, LBAs of bounce buffer 360-2 may beassigned or allocated to transaction contexts having TTags 610-1 to610-3. For these examples, the allocations may be similar to thosedescribed for bounce buffer 360-1 with the exception that rather thanincluding metadata with the given LBAs assigned to a give transactioncontext, a separate metadata area is reserved.

In some examples, an initial command may be received from a client thatincludes metadata. This initial command may set up a RDMA data transferbetween the client and a storage controlled by NVMe controller 350.Subsequent commands may then be received from the clients that do notinclude metadata. Since the subsequent commands lack metadata, theportion of a bounce buffer allocated to a transaction context may beused entirely for the data transfer rather than reserving a portion formetadata as described above for FIG. 5. Hence, by using the entireportion, bounce buffer 360-2 may be more efficiently used whensubsequent commands do not include metadata.

FIG. 7 illustrates an example Namespace context index 700. According tosome examples, Namespace context index 700 may include information usedto validate a command received from a client to provide the clientaccess to storage controlled by NVMe controller 350. As shown in FIG. 7,Namespace context index 700 depicts the various allocations orassignments of resources provided to clients 310-1 to 310-n describedabove for FIGS. 3-5. Additionally, Namespace context index 700 includesinformation to indicate what operational codes (Opcodes) are authorizedfor the clients to access the storage. These Opcodes for each client mayinclude, but are not limited to, one or more of flush (F), write (W),read (R), write uncorrectable (WU), compare (C).

According to some examples, the Opcodes authorized may also be based onwhat Opcodes are supported by the logic and/or features at NW I/O device330 to facilitate RDMA of the storage controlled by NVMe controller 350.For example, the NVME specification includes several Opcodes in additionto the five Opcodes listed above. However the additional Opcodes may befor operations unrelated or unnecessary for RDMA of the storagecontrolled by NVMe controller 350. Hence, commands including these otherOpcodes may be deemed as unsupported by the logic and/or features at NWI/O device 330.

In some examples, certain NVMe Namespace Identifiers may be limited asto the Opcodes allowed for those NVMe Namespace Identifiers. Forexample, NVMe Namespace Identifier 402 may be limited to only Opcodesfor read access. So as shown in FIG. 7 for Namespace context index 700,NSCs 21, 22 and 20 allocated to clients 310-1, 310-2 and 310-n,respectively have only an “R” for Opcode(s). Thus, if commands includingthese NSCs have Opcodes other than read Opcodes, those commands may notbe validated by logic and/or features at NW I/O device 330.

FIG. 8 illustrates an example communication flow 800. In some examples,as shown in FIG. 8, communication flow 800 depicts examplecommunications between client 310-1 and server 305. For these examples,the communications may be compatible with an RDMA (e.g., iWARP, IB,RoCE) protocol.

In some examples, starting from the top of FIG. 8, the first line “RDMAwrite (S-Stag) (Data)” may be an RDMA Write message carrying transactiondata. The second line “RDMA Write (S-Stag) (Out-of-Line Metadata)” maybe an RDMA Write carrying out-of-line Metadata, which may appear whenclient 310-1 is targeting a Namespace formatted with out-of-lineMetadata. The third line “RDMA Send (Command) (C-tag, S-Stag)” may be anRDMA Send message from client 310-1 that may include the command foraccess to the storage controlled by NVMe controller 350. The fourth line“RDMA Write (C-Stag) (Data)” may be an RDMA write message to carry datafrom a read of the storage and targeting a buffer (e.g., identified byC-Stag) maintained at client 310-1 that was originally indicated in aread request command. The fifth line “RDMA Send SE (Completion)” may bean RDMA Send message indicating that a solicited event (SE) such as aread command was completed by NVMe controller 350. The last line “RDMARead (Out-of-Line Metadata)” may be an RDMA Read message used if client310-1 was targeting a Namespace formatted with an out-of-line Metadata.

According to some examples, as shown in FIG. 8, solid lines may berelated to all Opcodes, dotted lines may be related to read only Opcodesand dashed lines may be related to write only Opcodes that may includewrite or compare commands.

FIG. 9 illustrates an example read command format 900. According to someexamples, messages in an RDMA (e.g., iWARP, IB, RoCE) protocol may besent from remote clients to a server using example read command format900. For these examples, clients 310-1 to 310-n may use example readcommand format 900 to relay a read command to access storage controlledby NVMe controller 350 at server 305. Logic and/or features at NW I/Odevice 330 may receive the messages and compare information included inthe messages to Namespace context index 700 in order to validate theread command. For example, the command may be validated based at leastin part, on whether the “Opcode” indicated in bit range [7:0] issupported by NW I/O device 330 and on whether the “NSC” indicated in bitrange [63:32] associates the sending client with an NVMe NamespaceIdentifier allocated to that client. The “Bounce Buffer TTag” indicatedin bit range [31:0] may be compared to information in Namespace contextindex 700 to validate whether the sending client has been allocated thetransaction context associated with that TTag. Further, “Starting LBA”indicated in bit range [63:0] and the “Number of Logical Blocks”indicated in bit range [15:0] may be used to determine the size of anaccess transaction for the command and then compare that size to theamount of memory capacity from bounce buffer 360-1 that was allocatedfor the transaction context associated with the “Bounce Buffer TTag”.The logic and/or features at NW I/O device 330 may invalidate the readcommand if the determined size exceeds the amount of memory capacityallocated.

This disclosure is not limited to the above mentioned examples ofinformation included in a read command received in the example format ofread command format 900. Other information included in a read commandeither in the example format of read command format 900 or other readcommand formats may be used to validate a read command received from aclient.

FIG. 10 illustrates an example write and compare command format 1000.According to some examples, messages in an RDMA (e.g., iWARP, IB, RoCE)protocol may be sent from remote clients to a server using write andcompare command format 1000. For these examples, clients 310-1 to 310-nmay use example write and compare command format 1000 to relay a writeor a compare command to access storage controlled by NVMe controller 350at server 305. Logic and/or features at NW I/O device 330 may receivethe messages and compare information included in the messages toNamespace context index 700 in order to validate the write or comparecommand. Information indicated in similar bit ranges as mentioned abovefor example read command format 900 may also be indicated in examplewrite and compare command format 1000. Thus, write or compare commandsreceived in the example format of write and compare command format 1000may be validated in a similar manner as mentioned above for FIG. 9.

This disclosure is not limited to the above mentioned examples ofinformation included in a write or compare command received in theexample format of write and compare command format 1000. Otherinformation included in a write or compare command either in the exampleformat of write and compare command format 1000 or other write orcompare command formats may be used to validate write or comparecommands received from a client.

FIG. 11 illustrates an example flush and write uncorrectable commandformat 1100. According to some examples, messages in an RDMA (e.g.,iWARP, IB, RoCE) protocol may be sent from remote clients to a serverusing flush and write uncorrectable command format 1100. For theseexamples, clients 310-1 to 310-n may use example flush and writeuncorrectable command format 1100 to relay a flush or a writeuncorrectable command to access storage controlled by NVMe controller350 at server 305. Logic and/or features at NW I/O device 330 mayreceive the messages and compare information included in the messages toNamespace context index 700 in order to validate the flush or writeuncorrectable command. Information indicated in similar bit ranges asmentioned above for example read command format 900 may also beindicated in example flush and write uncorrectable command format 1100.Thus, flush or write uncorrectable commands received in the exampleformat of flush and write uncorrectable command format 1100 may bevalidated in a similar manner as mentioned above for FIG. 9.

This disclosure is not limited to the above mentioned examples ofinformation included in a flush or write uncorrectable command receivedin the example format of flush and write uncorrectable command format1100. Other information included in a flush or write uncorrectablecommand either in the example format of flush and write uncorrectablecommand format 1100 or other flush or write correctable command formatsmay be used to validate flush or write correctable commands receivedfrom a client.

FIG. 12 illustrates an example NVMe command completion format 1200.According to some examples, logic and/or features at NW I/O device 330may be capable of sending a message in an RDMA (e.g., iWARP, IB, RoCE)protocol to a client to indicate a status of a received command from theclient. For these examples, the message may be sent using example NVMecommand completion format 1200. The “Status Field” of NVMe commandcompletion format 1200, for example, may indicate in bit range [63:49]the status of the received command. Also, the client may use “TTag IndexLow” information included in bit range [47:32] to associate a receivedcommand in the example NVMe command completion format 1200 with apreviously sent command having a TTag that matches at least some of theinformation included in bit range [47:32].

FIG. 13 illustrates an example NVMe command validation table 1300. Insome examples, NVMe command validation table 1300 may indicate thevarious validation checks that logic and/or features of NW I/O device330 may be capable of conducting to verify a command received from aclient. As shown in FIG. 13, the checks performed may vary based on theOpcodes indicated or included in the received commands. According tosome examples, information included in a received command (e.g., informat 900, 1000 or 1100) may be compared to information maintained inan index such as Namespace context index 700 in order to perform thechecks.

FIG. 14 illustrates an example block diagram of an apparatus 1400.Although apparatus 1400 shown in FIG. 14 has a limited number ofelements in a certain topology, it may be appreciated that the apparatus1400 may include more or less elements in alternate topologies asdesired for a given implementation.

The apparatus 1400 may be supported by circuitry 1420 maintained at anetwork I/O device coupled to a server. Circuitry 1420 may be arrangedto execute one or more software or firmware implemented modules 1422-a.It is worthy to note that “a” and “b” and “c” and similar designators asused herein are intended to be variables representing any positiveinteger. Thus, for example, if an implementation sets a value for a=5,then a complete set of software or firmware for modules 1422-a mayinclude modules 1422-1, 1422-2, 1422-3, 1422-4 or 1422-5. The examplespresented are not limited in this context and the different variablesused throughout may represent the same or different integer values.

According to some examples, circuitry 1420 may include a processor orprocessor circuitry. The processor or processor circuitry can be any ofvarious commercially available processors, including without limitationan AMD® Athlon®, Duron® and Opteron® processors; ARM® application,embedded and secure processors; IBM® and Motorola® DragonBall® andPowerPC® processors; IBM and Sony® Cell processors; Intel® Atom®,Celeron®, Core (2) Duo®, Core i3, Core i5, Core i7, Itanium®, Pentium®,Xeon®, Xeon Phi® and XScale® processors; and similar processors.According to some examples circuitry 1420 may also be an applicationspecific integrated circuit (ASIC) and at least some modules 1422-a maybe implemented as hardware elements of the ASIC.

According to some examples, apparatus 1400 may include a receive module1422-1. Receive module 1422-1 may be capable of receiving commands(s)1410 via messages in an RDMA compliant (e.g., iWARP, IB, RoCE) protocol.Commands(s) 1410 may have been sent from remote clients to a server. Forthese examples, the server may be coupled to a NW I/O device having anapparatus 1400. Command(s) 1410 may include commands to access storagecontrolled by an NVMe controller located at or with the server. Receivemodule 1422-1 may be capable of at least temporarily storing protocolinformation 1424-a (e.g., in a data structure such as a lookup table(LUT)) in order to interpret or decode at least portions of command(s)1410.

In some examples, apparatus 1400 may also include a validation module1422-2. Validation module 1422-2 may be capable of validating command(s)1410 received by receive module 1422-1. Validation module 1422-2 may useinformation included in an index (e.g., Namespace context index 700) toperform the validation. Validation module 1422-2 may be capable of atleast temporarily storing the index information in a data structure(e.g., an LUT). The index information is shown in FIG. 14 as NSC indexinformation 1426-b. Validation module 1422-2 may also perform variousvalidation checks using information at least temporarily stored in thedata structure (e.g., NVMe command validation table 1300). The checkinformation is shown in FIG. 14 as NVMe command check information1428-c. For these examples, validation module 1422-2 may perform thevalidation based, at least in part, on information included incommand(s) 1410 as compared to NSC index information 1426-b followingthe checks described in NVMe command check information 1428-c.

In some examples, apparatus 1400 may also include a payload module1422-3. Payload module 1422-3 may be capable of obtaining (e.g., from aLUT maintained by validation module 1422-2) information for placingwrite or compare data to a bounce buffer indicated in a validatedcommand. Payload module 1422-3 may use information from NSC indexinformation 1426-b to determine what LBAs of the bounce buffer to writethe data.

According to some examples, apparatus 1400 may also include a submissionmodule 1422-4. Submission module 1422-4 may be capable of writing avalidated command to an NVMe submission queue. Submission module 1422-4may use information from NSC index information 1426-b to determine whichNVMe submission queue to write the validated command(s) 1410.

In some examples, apparatus 1400 may also include a report module1422-5. Report module 1422-5 may be capable of indicating a status ofthe received command(s) 1410. The status, for example, may includeinvalid command 1440 if the command(s) 1410 were found be invalid.Completed command 1450 may be indicated if validated command(s) 1410were executed by the NVMe controller. Report module 1422-5 may becapable of at least temporarily storing protocol information 1424-a(e.g., in an LUT) in order to encode at least portions of invalidcommand 1440 or completed command 1450 in an RDMA compliant (e.g.,iWARP, IB, RoCE) protocol to be sent to the client that sent command(s)1410.

Included herein is a set of logic flows representative of examplemethodologies for performing novel aspects of the disclosedarchitecture. While, for purposes of simplicity of explanation, the oneor more methodologies shown herein are shown and described as a seriesof acts, those skilled in the art will understand and appreciate thatthe methodologies are not limited by the order of acts. Some acts may,in accordance therewith, occur in a different order and/or concurrentlywith other acts from that shown and described herein. For example, thoseskilled in the art will understand and appreciate that a methodologycould alternatively be represented as a series of interrelated states orevents, such as in a state diagram. Moreover, not all acts illustratedin a methodology may be required for a novel implementation.

A logic flow may be implemented in software, firmware, and/or hardware.In software and firmware embodiments, a logic flow may be implemented bycomputer executable instructions stored on at least one non-transitorycomputer readable medium or machine readable medium, such as an optical,magnetic or semiconductor storage. The embodiments are not limited inthis context.

FIG. 15 illustrates an example of a logic flow 1500. Logic flow 1500 maybe representative of some or all of the operations executed by one ormore logic, features, or devices described herein, such as apparatus1400. More particularly, logic flow 1500 may be implemented by receivemodule 1422-1, validation module 1422-2, payload module 1422-3,submission module 1422-4 or report module 1422-5.

According to some examples, logic flow 1500 at block 1502 may receive acommand from a client to provide the client access to a storage devicecontrolled by an NVMe controller maintained at a server. For example,command(s) 1410 may be received by receive module 1422-1 included in anapparatus 1400 for a NW I/O device coupled to the server.

In some examples, logic flow 1500 at block 1504 may then validate thecommand based, at least in part, on whether an Opcode included in thecommand is supported by the NW I/O device and on whether an indexidentifier included in the command associates the client with an NVMeNamespace Identifier allocated to the client. For example, validationmodule 1422-2 may compare information included in command(s) 1410 to NSCindex information 1426-b according to NVMe command check information1428-c to determine whether command(s) 1410 are valid.

According to some examples, logic flow 1500 at block 1506 may thenindicate a status of the received command to the client. For theseexamples, the status may include one of an invalid command or acompleted command. Report module 1422-5, for example, may indicateinvalid command 1440 to the client that sent command(s) 1410 ifvalidation module 1422-2 determined the command was invalid.Alternatively, report module 1422-5 may indicate that valid command(s)1410 were executed by the NVMe controller maintained at the server. Thatindication may be completed command 1450 and as mentioned above, may beindicated in a command completion queue assigned to the client that sentcommand(s) 1410.

FIG. 16 illustrates an example of a storage medium 1600. Storage medium1600 may comprise an article of manufacture. In some examples, storagemedium 1600 may include any non-transitory computer readable medium ormachine readable medium, such as an optical, magnetic or semiconductorstorage. Storage medium 1600 may store various types of computerexecutable instructions, such as instructions to implement logic flow1500. Examples of a computer readable or machine readable storage mediummay include any tangible media capable of storing electronic data,including volatile memory or non-volatile memory, removable ornon-removable memory, erasable or non-erasable memory, writeable orre-writeable memory, and so forth. Examples of computer executableinstructions may include any suitable type of code, such as source code,compiled code, interpreted code, executable code, static code, dynamiccode, object-oriented code, visual code, and the like. The examples arenot limited in this context.

FIG. 17 illustrates an example NW I/O device 1700. In some examples, asshown in FIG. 17, NW I/O device 1700 may include a processing component1740, other platform components or a communications interface 1760.According to some examples, network I/O device 1700 may be implementedin a NW I/O device coupled to a server in a system or data center asmentioned above.

According to some examples, processing component 1740 may executeprocessing operations or logic for apparatus 1400 and/or storage medium1600. Processing component 1740 may include various hardware elements,software elements, or a combination of both. Examples of hardwareelements may include devices, logic devices, components, processors,microprocessors, circuits, processor circuits, circuit elements (e.g.,transistors, resistors, capacitors, inductors, and so forth), integratedcircuits, application specific integrated circuits (ASIC), programmablelogic devices (PLD), digital signal processors (DSP), field programmablegate array (FPGA), memory units, logic gates, registers, semiconductordevice, chips, microchips, chip sets, and so forth. Examples of softwareelements may include software components, programs, applications,computer programs, application programs, device drivers, systemprograms, software development programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Determining whether an example isimplemented using hardware elements and/or software elements may vary inaccordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints, as desired for a given example.

In some examples, other platform components 1750 may include commoncomputing elements, such as one or more processors, multi-coreprocessors, co-processors, memory units, chipsets, controllers,peripherals, interfaces, oscillators, timing devices, and so forth.Examples of memory units may include without limitation various types ofcomputer readable and machine readable storage media in the form of oneor more higher speed memory units, such as ROM, RAM, DRAM, DDRAM, SDRAM,SRAM, PROM, EPROM, EEPROM, flash memory or any other type of storagemedia suitable for storing information.

In some examples, communications interface 1760 may include logic and/orfeatures to support a communication interface. For these examples,communications interface 1760 may include one or more communicationinterfaces that operate according to various communication protocols orstandards to communicate over direct or network communication links.Direct communications may occur via use of communication protocols orstandards described in one or more industry standards (includingprogenies and variants) such as those associated with the PCIespecification, the NVMe specification, the RDMA Protocol specification,the IEEE 802-2-2008 specification, RFC 791 or RFC 793.

The components and features of NW I/O device 1700 may be implementedusing any combination of discrete circuitry, application specificintegrated circuits (ASICs), logic gates and/or single chiparchitectures. Further, the features of NW I/O device 1700 may beimplemented using microcontrollers, programmable logic arrays and/ormicroprocessors or any combination of the foregoing where suitablyappropriate. It is noted that hardware, firmware and/or softwareelements may be collectively or individually referred to herein as“logic” or “circuit.”

It should be appreciated that the exemplary NW I/O device 1700 shown inthe block diagram of FIG. 17 may represent one functionally descriptiveexample of many potential implementations. Accordingly, division,omission or inclusion of block functions depicted in the accompanyingfigures does not infer that the hardware components, circuits, softwareand/or elements for implementing these functions would necessarily bedivided, omitted, or included in embodiments.

One or more aspects of at least one example may be implemented byrepresentative instructions stored on at least one machine-readablemedium which represents various logic within the processor, which whenread by a machine, computing device or system causes the machine,computing device or system to fabricate logic to perform the techniquesdescribed herein. Such representations, known as “IP cores” may bestored on a tangible, machine readable medium and supplied to variouscustomers or manufacturing facilities to load into the fabricationmachines that actually make the logic or processor.

Various examples may be implemented using hardware elements, softwareelements, or a combination of both. In some examples, hardware elementsmay include devices, components, processors, microprocessors, circuits,circuit elements (e.g., transistors, resistors, capacitors, inductors,and so forth), integrated circuits, application specific integratedcircuits (ASIC), programmable logic devices (PLD), digital signalprocessors (DSP), field programmable gate array (FPGA), memory units,logic gates, registers, semiconductor device, chips, microchips, chipsets, and so forth. In some examples, software elements may includesoftware components, programs, applications, computer programs,application programs, system programs, machine programs, operatingsystem software, middleware, firmware, software modules, routines,subroutines, functions, methods, procedures, software interfaces,application program interfaces (API), instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof. Determining whether an example isimplemented using hardware elements and/or software elements may vary inaccordance with any number of factors, such as desired computationalrate, power levels, heat tolerances, processing cycle budget, input datarates, output data rates, memory resources, data bus speeds and otherdesign or performance constraints, as desired for a givenimplementation.

Some examples may include an article of manufacture or at least onecomputer-readable medium. A computer-readable medium may include anon-transitory storage medium to store logic. In some examples, thenon-transitory storage medium may include one or more types ofcomputer-readable storage media capable of storing electronic data,including volatile memory or non-volatile memory, removable ornon-removable memory, erasable or non-erasable memory, writeable orre-writeable memory, and so forth. In some examples, the logic mayinclude various software elements, such as software components,programs, applications, computer programs, application programs, systemprograms, machine programs, operating system software, middleware,firmware, software modules, routines, subroutines, functions, methods,procedures, software interfaces, API, instruction sets, computing code,computer code, code segments, computer code segments, words, values,symbols, or any combination thereof.

According to some examples, a computer-readable medium may include anon-transitory storage medium to store or maintain instructions thatwhen executed by a machine, computing device or system, cause themachine, computing device or system to perform methods and/or operationsin accordance with the described examples. The instructions may includeany suitable type of code, such as source code, compiled code,interpreted code, executable code, static code, dynamic code, and thelike. The instructions may be implemented according to a predefinedcomputer language, manner or syntax, for instructing a machine,computing device or system to perform a certain function. Theinstructions may be implemented using any suitable high-level,low-level, object-oriented, visual, compiled and/or interpretedprogramming language.

Some examples may be described using the expression “in one example” or“an example” along with their derivatives. These terms mean that aparticular feature, structure, or characteristic described in connectionwith the example is included in at least one example. The appearances ofthe phrase “in one example” in various places in the specification arenot necessarily all referring to the same example.

Some examples may be described using the expression “coupled” and“connected” along with their derivatives. These terms are notnecessarily intended as synonyms for each other. For example,descriptions using the terms “connected” and/or “coupled” may indicatethat two or more elements are in direct physical or electrical contactwith each other. The term “coupled,” however, may also mean that two ormore elements are not in direct contact with each other, but yet stillco-operate or interact with each other.

It is emphasized that the Abstract of the Disclosure is provided tocomply with 37 C.F.R. Section 1.72(b), requiring an abstract that willallow the reader to quickly ascertain the nature of the technicaldisclosure. It is submitted with the understanding that it will not beused to interpret or limit the scope or meaning of the claims. Inaddition, in the foregoing Detailed Description, it can be seen thatvarious features are grouped together in a single example for thepurpose of streamlining the disclosure. This method of disclosure is notto be interpreted as reflecting an intention that the claimed examplesrequire more features than are expressly recited in each claim. Rather,as the following claims reflect, inventive subject matter lies in lessthan all features of a single disclosed example. Thus the followingclaims are hereby incorporated into the Detailed Description, with eachclaim standing on its own as a separate example. In the appended claims,the terms “including” and “in which” are used as the plain-Englishequivalents of the respective terms “comprising” and “wherein,”respectively. Moreover, the terms “first,” “second,” “third,” and soforth, are used merely as labels, and are not intended to imposenumerical requirements on their objects.

According to some examples, an example apparatus may include circuitryfor a network input/output device coupled to a server. The exampleapparatus may also include a receive module for execution by thecircuitry to receive a command from a client remote to the server. Thecommand may provide the client access to a storage device controlled bya Non-Volatile Memory Express (NVMe) controller maintained at theserver. The example apparatus may also include a validation module forexecution by the circuitry to validate the command based, at least inpart, on whether an operation code (Opcode) indicated in the command issupported by the network input/output device and on whether an indexidentifier indicated in the command associates the client with an NVMeNamespace Identifier allocated to the client. The example apparatus mayalso include a report module for execution by the circuitry to indicatea status of the received command to the client. The status may includeone of an invalid command or a completed command.

In some examples, the example apparatus may also include the networkinput/output device, the storage device and the NVMe controller beingarranged to operate in compliance with an industry standard to includePCIe Base Specification, revision 3.0 or NVMe Specification, revision1.1.

According to some examples for example apparatus, the command may bereceived in a packet compatible with a remote direct memory access(RDMA) protocol to include one of Internet Wide Area RDMA protocol(iWARP), Infinband or RDMA over Converged Ethernet (RoCE).

In some examples for the example apparatus, the Opcode included in thecommand including one of a flush, a write, a read, a write uncorrectableor a compare.

According to some examples, the example apparatus may also include thestorage device including a hard disk drive (HDD) or a solid state drive(SSD). The SSD may have non-volatile memory that includes at least oneof 3-dimensional cross-point memory, flash memory, ferroelectric memory,silicon-oxide-nitride-oxide-silicon (SONOS) memory, polymer memory,nanowire, ferroelectric transistor random access memory (FeTRAM orFeRAM), nanowire or electrically erasable programmable read-only memory(EEPROM).

In some examples for the example apparatus, the index identifier thatassociates the client with the Namespace Identifier is included in anindex that indicates allocated resources provided to the client by theserver for access to the storage device, the allocated resources toinclude a buffer maintained by the server and a first queue pairaccessible to the client that includes a command submission queue to theNVMe controller and a command completion queue from the NVMe controller

According to some examples for the example apparatus, the allocatedresources may also include a second queue pair accessible to the NVMecontroller that includes a second command submission queue to the NVMecontroller and a second command completion queue from the NVMecontroller.

In some examples, the example apparatus may also include the reportmodule capable of indicating the status of the received command as acompleted command via an indication in the first command completionqueue that indicates the received command to access the storage devicewas executed by the NVMe controller.

According to some examples for the example apparatus, the allocatedresources may include a portion of a memory capacity for the buffer. Thevalidation module may also validate the command based on whether atransaction size to access the storage device indicated in the commandexceeds the portion of the memory capacity.

In some examples for the example apparatus, the index identifier thatassociates the client with the NVMe Namespace Identifier may alsoassociate the client with a first transaction context assigned to thebuffer. The validation module may also validate the command based onwhether a second transaction context indicated in the received commandmatches the first transaction context.

According to some examples for the example apparatus, the indexidentifier that associates the client with the NVMe Namespace Identifiermay also associate the NVMe Namespace Identifier with an Opcode. Thevalidation module may also validate the command based on whether theOpcode indicated in the command is associated with the NVMe NamespaceIdentifier.

In some examples, example methods may include receiving, at a networkinput/output device coupled to a server, a command for a client remoteto the server to access a storage device controlled by a Non-VolatileMemory Express (NVMe) controller maintained at the server. The commandmay be validated based on whether an operation code (Opcode) indicatedin the command is supported by the network input/output device and onwhether an index identifier indicated in the command associates theclient with an NVMe Namespace Identifier allocated to the client. Astatus of the received command may then be indicated to the client asone of an invalid command or a completed command.

According to some examples, the example methods may also include thenetwork input/output device, the storage device and the NVMe controllerbeing arranged to operate in compliance with an industry standard toinclude PCIe Base Specification, revision 3.0 or NVMe Specification,revision 1.1.

In some examples for the example methods, the allocated resources toalso include a second queue pair accessible to the NVMe controller thatincludes a second command submission queue to the NVMe controller and asecond command completion queue from the NVMe controller.

According to some examples for the example methods, the status of thereceived command may be indicated as a completed command via acompletion indication placed in the first command completion queue. Thecompletion indication may indicate that the received command to accessthe storage device was executed by the NVMe controller.

In some examples for the example methods, the index that associates theclient with the NVMe Namespace Identifier may also associate the clientwith a first transaction context assigned to the buffer. Validating thecommand may also include validating the command based on whether asecond transaction context indicated in the received command matches thefirst transaction context.

According to some examples for the example methods, the storage deviceto include a hard disk drive (HDD) or a solid state drive (SSD). Forthese examples, the SSD may have non-volatile memory that includes atleast one of 3-dimensional cross-point memory, flash memory,ferroelectric memory, silicon-oxide-nitride-oxide-silicon (SONOS)memory, polymer memory, nanowire, ferroelectric transistor random accessmemory (FeTRAM or FeRAM), nanowire or electrically erasable programmableread-only memory (EEPROM).

According to some examples, at least one machine readable mediumcomprising a plurality of instructions that in response to beingexecuted on a network input/output device coupled to a server cause thenetwork I/O device to carry out the example method as mentioned above.

Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the specific features or acts described above.Rather, the specific features and acts described above are disclosed asexample forms of implementing the claims.

What is claimed is:
 1. An apparatus comprising: circuitry for a remotedirect memory access (RDMA)-enabled network input/output device coupledto a server; logic for execution by the circuitry to: allocate, in amemory of the server, a buffer, a doorbell, and a queue pair accessibleto a client remote to the server, the queue pair comprising a commandsubmission queue to a Non-Volatile Memory Express (NVMe) controllermaintained at the server and a command completion queue from the NVMecontroller, the NVMe controller to provide the client access to astorage device controlled by the NVMe controller; assign an NVMenamespace context to the client; and transmit, to the client,indications of the buffer, the doorbell, the command submission queue,the command completion queue, and the NVMe namespace context.
 2. Theapparatus of claim 1, the logic for execution by the circuitry to:receive a command from the client via a network, the command comprisingan operation code (Opcode) and an index identifier; validate the commandbased, at least in part, on whether the Opcode is supported by thenetwork input/output device, on whether the index identifier matches theNVMe Namespace context, and whether the NVMe namespace contextassociates the client with the NVMe Namespace Identifier allocated tothe client; and write a value specified by the command to the doorbell.3. The apparatus of claim 2, the logic for execution by the circuitryto: execute, by the NVMe controller against the storage, the commandbased on the value written to the doorbell.
 4. The apparatus of claim 2,the network input/output device, the storage device, and the NVMecontroller arranged to operate in compliance with PCIe BaseSpecification, revision 3.0 or NVMe Specification, revision 1.1, thecommand received in a packet compatible with a remote direct memoryaccess (RDMA) protocol to include one of Internet Wide Area RDMAprotocol (iWARP), Infinband or RDMA over Converged Ethernet (RoCE). 5.The apparatus of claim 2, the value written to the doorbell based atleast in part on one or more Message Signaled Interrupts.
 6. Theapparatus of claim 1, the indications to comprise a respective addressof the buffer, the doorbell, the command submission queue, and thecommand completion queue in the memory.
 7. The apparatus of claim 1, theindications to comprise one or more of service tags (STags) ortransaction tags (TTags) compatible with a remote direct memory access(RDMA) protocol to include one of Internet Wide Area RDMA protocol(iWARP), Infinband or RDMA over Converged Ethernet (RoCE).
 8. Theapparatus of claim 1, the client one of a plurality of clients, thelogic configured to allocate, to each client, a respective buffer, arespective doorbell, an NVMe namespace context, and a respective queuepair comprising a command submission queue and a command completionqueue.
 9. The apparatus of claim 1, the client one of a plurality ofclients, the buffer, the command submission queue, the commandcompletion queue, and the doorbell shared by the plurality of clients,the logic to allocate a respective portion of the buffer, the commandsubmission queue, the command completion queue, and the doorbell to eachclient.
 10. The apparatus of claim 1, the client to comprise anapplication to access the storage via an RDMA-enabled networkinput/output device of the client.
 11. The apparatus of claim 1, thecommand completion queue to comprise an asynchronous queue, the logic tostore command completion queue elements in the command completion queuebased at least in part on at least one of: (i) an order in which commandqueue elements corresponding to the command completion queue elementsare stored in the command submission queue, (ii) an order of executionof the command queue elements, or (iii) an order in which the completionqueue elements are provided to the client.
 12. A method, comprising:allocating, in a memory of a server by a remote direct memory access(RDMA)-enabled network input/output device coupled to the server, abuffer, a doorbell, and a queue pair accessible to a client remote tothe server, the queue pair comprising a command submission queue to aNon-Volatile Memory Express (NVMe) controller maintained at the serverand a command completion queue from the NVMe controller, the NVMecontroller to provide the client access to a storage device controlledby the NVMe controller; assigning an NVMe namespace context to theclient; and transmitting, to the client, indications of the buffer, thedoorbell, the command submission queue, the command completion queue,and the NVMe namespace context.
 13. The method of claim 12, comprising:receiving, by the network input/output device, a command from the clientvia a network, the command comprising an operation code (Opcode) and anindex identifier; validating the command, at least in part, on whetherthe Opcode is supported by the network input/output device, on whetherthe index identifier matches the NVMe Namespace context, and whether theNVMe namespace context associates the client with the NVMe NamespaceIdentifier allocated to the client; and writing a value specified by thecommand to the doorbell.
 14. The method of claim 13, comprising:executing, by the NVMe controller against the storage, the command basedon the writing of the value to the doorbell, the value written to thedoorbell based at least in part on one or more Message SignaledInterrupts.
 15. The method of claim 13, the network input/output device,the storage device, and the NVMe controller arranged to operate incompliance with PCIe Base Specification, revision 3.0 or NVMeSpecification, revision 1.1, the command received in a packet compatiblewith a remote direct memory access (RDMA) protocol to include one ofInternet Wide Area RDMA protocol (iWARP), Infinband or RDMA overConverged Ethernet (RoCE).
 16. The method of claim 12, the indicationsto comprise one or more of service tags (STags) or transaction tags(TTags) compatible with a remote direct memory access (RDMA) protocol toinclude one of Internet Wide Area RDMA protocol (iWARP), Infinband orRDMA over Converged Ethernet (RoCE).
 17. The method of claim 12, theclient one of a plurality of clients, the method comprising allocating,to each client, a respective buffer, a respective doorbell, an NVMenamespace context, and a respective queue pair comprising a commandsubmission queue and a command completion queue.
 18. The method of claim12, the client one of a plurality of clients, the buffer, the commandsubmission queue, the command completion queue, and the doorbell sharedby the plurality of clients, the method comprising allocating arespective portion of the buffer, the command submission queue, thecommand completion queue, and the doorbell to each client.
 19. Anon-transitory machine-readable storage medium comprising instructionswhich when executed by processing circuitry of a server cause theprocessing circuitry to: allocate, in a memory of the server by a remotedirect memory access (RDMA)-enabled network input/output device coupledto the server, a buffer, a doorbell, and a queue pair accessible to aclient remote to the server, the queue pair comprising a commandsubmission queue to a Non-Volatile Memory Express (NVMe) controllermaintained at the server and a command completion queue from the NVMecontroller, the NVMe controller to provide the client access to astorage device controlled by the NVMe controller; assign an NVMenamespace context to the client; and transmit, to the client,indications of the buffer, the doorbell, the command submission queue,the command completion queue, and the NVMe namespace context.
 20. Themachine-readable storage medium of claim 19, comprising instructionswhich when executed by the processing circuitry cause the processingcircuitry to: receive a command from the client via a network, thecommand comprising an operation code (Opcode) and an index identifier;validate the command based, at least in part, on whether the Opcode issupported by the network input/output device, on whether the indexidentifier matches the NVMe Namespace context, and whether the NVMenamespace context associates the client with the NVMe NamespaceIdentifier allocated to the client; and write a value specified by thecommand to the doorbell.
 21. The machine-readable storage medium ofclaim 20, the network input/output device, the storage device, and theNVMe controller arranged to operate in compliance with PCIe BaseSpecification, revision 3.0 or NVMe Specification, revision 1.1, thecommand received in a packet compatible with a remote direct memoryaccess (RDMA) protocol to include one of Internet Wide Area RDMAprotocol (iWARP), Infinband or RDMA over Converged Ethernet (RoCE). 22.The machine-readable storage medium of claim 20, the value written tothe doorbell based at least in part on one or more Message SignaledInterrupts.
 23. The machine-readable storage medium of claim 19, theindications to comprise one or more of service tags (STags) ortransaction tags (TTags) compatible with a remote direct memory access(RDMA) protocol to include one of Internet Wide Area RDMA protocol(iWARP), Infinband or RDMA over Converged Ethernet (RoCE).
 24. Themachine-readable storage medium of claim 19, the client one of aplurality of clients, the buffer, the command submission queue, thecommand completion queue, and the doorbell shared by the plurality ofclients, the storage medium comprising instructions which when executedby the processing circuitry cause the processing circuitry to allocate arespective portion of the buffer, the command submission queue, thecommand completion queue, and the doorbell to each client.